DRAFT: Rebase av2enc first milestone on main - #5330
Conversation
|
Hi @urvangjoshi, @leolzhao, and @yunqingwang1, I've rebased on top of the main branch, stopping at the commit corresponding to Jianle's "Enable deblock filter for partition research" — the one we used to report the first milestone results. Meanwhile, I'm running the speed 0/1/4 tests on the rebased branch to confirm nothing is broken, and the results would look as expected. Feel free to review in parallel and let me know if anything needs changing.
Thanks |
|
@yeqing-wu : Thanks a lot for writing this PR! I did some overall sanity checks: (1) I see ~60 commits in this commit range on av2-enc branch, but 51 commits in this PR. I'm assuming other commits were already on main branch (previous cherry-picks)? |
|
@urvangjoshi, thanks very much for helping review the PR and your suggestion. Please see my reply.
Yes — 9 commits have already landed in main independently; see the table. Should we discourage experts from merging code into both main and the av2-enc branch unless it's a critical bug fix? Otherwise it becomes hard for us to track.
Sure, I will do it tomorrow.
Good catch — that reduces the risk behind the experts' concerns in VCWG.
Sure, I'll do it once I have a reasonable number of bitstreams. Thanks |
|
@urvangjoshi, regarding the email issues, I noticed that they exist even in the av2-enc branch. Thus, the issue is not caused by the rebase. I believe they might be utilizing Google’s internal systems and then submitting the code to av2-enc because it includes the
Here are the screenshots from the av2-enc branch.
|
|
In GitHub, from the PR, we cannot see the email address. |
Yes, what seems to have happened is that the PRs on av2-enc branch had correct email address (so CLA check passed), but when the PRs got merged, it used this e.g. #5080 had the correct email address
There is a trick to seeing the email: Now, coming back to this PR: Going forward, we'll have to ensure future av2-enc commits don't run into this issue.
|
Anchor: research-v15.0.0 CTC: v9 Baseline: speed 0 Configuration: RA 33frames (A2 - A5) Original (research-v15.0.0) Speed 1: PSNR-YUV: +0.66% | VMAF: +0.51% | EncTime: 71.38% Speed 2: PSNR-YUV: +4.46% | VMAF: +5.00% | EncTime: 31.91% Speed 3: PSNR-YUV: +13.00% | VMAF: +14.41% | EncTime: 16.95% Speed 4: PSNR-YUV: +16.86% | VMAF: +18.32% | EncTime: 13.32% Speed 5: PSNR-YUV: +21.24% | VMAF: +23.32% | EncTime: 9.53% New Speed 1: PSNR-YUV: +0.66% | VMAF: +0.51% | EncTime: 71.38% Speed 2: PSNR-YUV: +3.37% | VMAF: +3.51% | EncTime: 32.26% Speed 3: PSNR-YUV: +8.45% | VMAF: +9.50% | EncTime: 19.44% Speed 4: PSNR-YUV: +10.48% | VMAF: +11.55% | EncTime: 14.91% Speed 5: PSNR-YUV: +11.45% | VMAF: +12.80% | EncTime: 13.95% Configuration: RA 17frames (A1) Original (research-v15.0.0) Speed 1: PSNR-YUV: +0.42% | VMAF: +0.35% | EncTime: 82.13% Speed 2: PSNR-YUV: +5.96% | VMAF: +5.92% | EncTime: 40.46% Speed 3: PSNR-YUV: +15.80% | VMAF: +16.50% | EncTime: 25.52% Speed 4: PSNR-YUV: +20.19% | VMAF: +21.10% | EncTime: 22.01% Speed 5: PSNR-YUV: +25.91% | VMAF: +30.35% | EncTime: 16.90% New Speed 1: PSNR-YUV: +0.42% | VMAF: +0.35% | EncTime: 82.13% Speed 2: PSNR-YUV: +4.27% | VMAF: +4.07% | EncTime: 42.26% Speed 3: PSNR-YUV: +9.82% | VMAF: +10.11% | EncTime: 29.48% Speed 4: PSNR-YUV: +12.02% | VMAF: +12.67% | EncTime: 24.22% Speed 5: PSNR-YUV: +13.34% | VMAF: +13.82% | EncTime: 23.12%
These functions are only needed by the encoder.
Only assign pc_tree->partitioning = partition when last_part_rdc represents a valid RDO search result. Guard the do_recon block so reconstruction is only attempted on a valid partition. Fixes AOMediaCodec#5092 Co-authored-by: Urvang Joshi <urvang@google.com>
…5098) Drive the main inter-mode loop in av2_rd_pick_inter_mode_sb from a static ref-frame-centric LUT and revise the motion-mode pruning heuristic. STATS_CHANGED Anchor: commit 165345b Speed 1 (cpu-used=1): FG16 CTC (33 frames, class A1 and A2, RA) Speed 0 (cpu-used=0): VCWG CTC (130 frames, all classes, RA and LD). Speed 4 (cpu-used=4): FG16 CTC (33 frames, class A1 and A2, RA) ``` 1) This change in isolation (loop restructure + motion-mode pruning). Speed 1 (cpu-used=1), RA: +------------+------+-------+-------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+-------+-------+------+------+------+ | A1 | 0.18 | -0.03 | -0.20 | 0.14 | 95 | 101 | | A2 | 0.18 | 0.67 | 0.35 | 0.20 | 93 | 100 | | Avg w/o B2 | 0.18 | 0.47 | 0.19 | 0.18 | 94 | 100 | +------------+------+-------+-------+------+------+------+ Speed 0 (cpu-used=0), RA: +------------+-------+-------+-------+-------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+-------+-------+-------+-------+------+------+ | A1 | -0.01 | 0.05 | -0.05 | -0.01 | 100 | 101 | | A2 | 0.01 | 0.02 | 0.06 | 0.01 | 100 | 100 | | A3 | 0.00 | 0.01 | 0.10 | 0.01 | 100 | 100 | | A4 | -0.02 | 0.01 | -0.51 | -0.03 | 99 | 100 | | A5 | -0.01 | -0.50 | 0.05 | -0.03 | 101 | 100 | | B1 | 0.05 | 0.18 | -0.04 | 0.04 | 100 | 101 | | B2 | 0.36 | 0.36 | 0.32 | 0.36 | 101 | 108 | | Avg w/o B2 | 0.01 | 0.01 | -0.03 | 0.00 | 100 | 100 | +------------+-------+-------+-------+-------+------+------+ Speed 0 (cpu-used=0), LD: +------------+-------+-------+-------+-------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+-------+-------+-------+-------+------+------+ | A2 | 0.02 | -0.01 | -0.14 | 0.01 | 100 | 100 | | A3 | -0.03 | 0.01 | 0.63 | -0.01 | 101 | 101 | | A4 | -0.00 | -0.26 | -1.13 | -0.05 | 100 | 100 | | A5 | 0.16 | 0.60 | 0.05 | 0.16 | 102 | 103 | | B1 | 0.02 | -0.07 | -0.03 | 0.01 | 101 | 100 | | B2 | 0.36 | 0.34 | 0.61 | 0.38 | 101 | 102 | | Avg w/o B2 | 0.02 | 0.00 | -0.10 | 0.01 | 101 | 100 | +------------+-------+-------+-------+-------+------+------+ 2) For reference, the two follow-up commits (single-ref to compound pruning, and per-ref RD-based pruning for single-ref NEWMV / WARP_NEWMV) combined, Speed 1 (cpu-used=1), RA: +------------+------+-------+-------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+-------+-------+------+------+------+ | A1 | 0.20 | -0.08 | -0.06 | 0.16 | 83 | 102 | | A2 | 0.22 | 0.53 | 0.15 | 0.23 | 84 | 101 | | Avg w/o B2 | 0.22 | 0.35 | 0.09 | 0.21 | 84 | 101 | +------------+------+-------+-------+------+------+------+ 3) Overall, motion-mode pruning stacked on top of (2), Speed 1 (cpu-used=1), RA: +------------+------+------+-------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+------+-------+------+------+------+ | A1 | 0.31 | 0.29 | -0.20 | 0.28 | 80 | 101 | | A2 | 0.32 | 0.90 | 0.58 | 0.35 | 80 | 100 | | Avg w/o B2 | 0.32 | 0.72 | 0.35 | 0.33 | 80 | 100 | +------------+------+------+-------+------+------+------+ 4) Overall, motion-mode pruning, single-ref to compound pruning, and per-ref RD-based pruning for single-ref NEWMV / WARP_NEWMV combined, Speed 4 (cpu-used=4), RA: +------------+------+------+------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+------+------+------+------+------+ | A1 | 0.27 | 0.17 | 0.00 | 0.24 | 82 | 102 | | A2 | 0.42 | 0.63 | 0.15 | 0.42 | 81 | 102 | | Avg w/o B2 | 0.38 | 0.49 | 0.11 | 0.36 | 81 | 102 | +------------+------+------+------+------+------+------+ ```
The intra chroma component will not use FSC mode. Simplify the encoding functions therein.
Revive prune_compound_using_single_ref: refresh the per-ref RD snapshot every iteration and prune compound (refs[0], refs[1]) pairs via the single-ref RD cutoff. Enabled at speed >= 1. STATS_CHANGED Anchor: commit AOMediaCodec@165345b Speed 1 (cpu-used=1): FG16 CTC (33 frames, class A1 and A2, RA) ``` +------------+------+-------+-------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+-------+-------+------+------+------+ | A1 | 0.10 | -0.15 | -0.27 | 0.07 | 92 | 101 | | A2 | 0.09 | 0.28 | -0.12 | 0.08 | 93 | 101 | | Avg w/o B2 | 0.09 | 0.15 | -0.17 | 0.08 | 93 | 101 | +------------+------+-------+-------+------+------+------+ ```
…odec#5114) Tighten the loop lower bound to pb_mv_precision + 1 so the found cache precision is strictly greater than the current pb_mv_precision, matching the assert's intent.
…iaCodec#5100) Prune single-ref NEWMV / WARP_NEWMV trials whose prior-mode RD for the current ref frame is far from the best across all refs. Gated by prune_newmv_modes_using_prior_rd, enabled at speed >= 1. STATS_CHANGED Anchor: commit AOMediaCodec@165345b Speed 1 (cpu-used=1): FG16 CTC (33 frames, class A1 and A2, RA) ``` +------------+------+-------+-------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+-------+-------+------+------+------+ | A1 | 0.14 | -0.04 | -0.14 | 0.11 | 88 | 101 | | A2 | 0.17 | 0.66 | 0.10 | 0.18 | 88 | 101 | | Avg w/o B2 | 0.16 | 0.45 | 0.03 | 0.16 | 88 | 101 | +------------+------+-------+-------+------+------+------+ ```
Adds three single-ref NEWMV DRL search speed features, enabled at speed >= 1: * mv_sf.skip_second_best_subpel: the subpel search normally refines both the best and the second-best full-pel candidate and keeps the lower-variance result; with this feature set, only the best full-pel candidate is refined. * mv_sf.predict_repeated_newmv: when a previously searched ref_mv_idx has a reference MV within one full pel of the current one, reuse the cached NEWMV result and only recompute the MV rate, avoiding the full-pel and subpel search entirely. * mv_sf.newmv_drl_search_limit: once ref_mv_idx reaches the limit, reuse the nearest already-searched result regardless of ref-MV distance instead of running a fresh search. Enabled with a cap of 2. Shares the same search-loop pass as the predict_repeated_newmv reuse. STATS_CHANGED Anchor: commit 165345b Speed 1 (cpu-used=1): FG16 CTC (33 frames, class A1 and A2, RA) Speed 4 (cpu-used=4): FG16 CTC (33 frames, class A1 and A2, RA) ``` 1) Speed 1 a) Overall performance (all three features combined) +------------+------+-------+-------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+-------+-------+------+------+------+ | A1 | 0.04 | -0.08 | -0.36 | 0.00 | 99 | 101 | | A2 | 0.04 | -0.04 | -0.18 | 0.03 | 98 | 100 | | Avg w/o B2 | 0.04 | -0.05 | -0.23 | 0.02 | 98 | 100 | +------------+------+-------+-------+------+------+------+ b) Performance of skip_second_best_subpel only +------------+-------+-------+-------+-------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+-------+-------+-------+-------+------+------+ | A1 | -0.10 | -0.32 | 0.09 | -0.11 | 99.5 | 101 | | A2 | -0.01 | -0.09 | -0.34 | -0.02 | 99.5 | 100 | | Avg w/o B2 | -0.04 | -0.16 | -0.21 | -0.05 | 99.5 | 100 | +------------+-------+-------+-------+-------+------+------+ c) Performance of predict_repeated_newmv only +------------+-------+-------+-------+-------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+-------+-------+-------+-------+------+------+ | A1 | 0.00 | -0.08 | 0.08 | 0.00 | 100 | 100 | | A2 | -0.03 | -0.04 | -0.04 | -0.03 | 100 | 100 | | Avg w/o B2 | -0.02 | -0.05 | -0.00 | -0.02 | 100 | 100 | +------------+-------+-------+-------+-------+------+------+ d) Performance of newmv_drl_search_limit only +------------+------+-------+------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+-------+------+------+------+------+ | A1 | 0.05 | -0.17 | 0.09 | 0.03 | 99 | 101 | | A2 | 0.00 | 0.36 | 0.01 | 0.02 | 99 | 100 | | Avg w/o B2 | 0.01 | 0.20 | 0.03 | 0.02 | 99 | 100 | +------------+------+-------+------+------+------+------+ 2) Speed 4 Overall performance (all three features combined) +------------+------+-------+-------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+-------+-------+------+------+------+ | A1 | 0.01 | -0.13 | 0.07 | 0.00 | 96 | 101 | | A2 | 0.05 | -0.13 | -0.25 | 0.03 | 96 | 100 | | Avg w/o B2 | 0.04 | -0.13 | -0.15 | 0.02 | 96 | 100 | +------------+------+-------+-------+------+------+------+ ```
Enables three ref-index based pruning speed features at speed >= 1: - prune_refinemv_by_ref_idx: skip refinemv_loop == 1 for ref-pairs other than (0, 1). - prune_interintra_by_ref_idx: skip INTERINTRA motion mode when ref_frame[0] > 1. - prune_warp_delta_by_ref_idx: skip WARP_DELTA motion mode when ref_frame[0] > 2. STATS_CHANGED Anchor: commit 165345b Speed 1 (cpu-used=1): FG16 CTC (33 frames, class A1 and A2, RA) Speed 4 (cpu-used=4): FG16 CTC (33 frames, class A1 and A2, RA) ``` 1) Speed 1 a) Overall performance (all three features combined) +------------+------+-------+-------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+-------+-------+------+------+------+ | A1 | 0.02 | -0.15 | 0.05 | 0.01 | 93 | 101 | | A2 | 0.12 | 0.36 | -0.03 | 0.12 | 93 | 100 | | Avg w/o B2 | 0.09 | 0.21 | -0.00 | 0.09 | 93 | 100 | +------------+------+-------+-------+------+------+------+ b) Performance of prune_refinemv_by_ref_idx only +------------+-------+-------+-------+-------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+-------+-------+-------+-------+------+------+ | A1 | -0.01 | -0.14 | -0.34 | -0.04 | 99 | 101 | | A2 | 0.04 | 0.06 | -0.25 | 0.02 | 99 | 100 | | Avg w/o B2 | 0.02 | 0.00 | -0.28 | 0.00 | 99 | 100 | +------------+-------+-------+-------+-------+------+------+ c) Performance of prune_interintra_by_ref_idx only +------------+------+-------+------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+-------+------+------+------+------+ | A1 | 0.04 | -0.02 | 0.04 | 0.04 | 98 | 100 | | A2 | 0.02 | 0.27 | 0.08 | 0.03 | 98 | 100 | | Avg w/o B2 | 0.03 | 0.18 | 0.07 | 0.03 | 98 | 100 | +------------+------+-------+------+------+------+------+ d) Performance of prune_warp_delta_by_ref_idx only +------------+------+------+-------+------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+------+------+-------+------+------+------+ | A1 | 0.03 | 0.00 | 0.12 | 0.03 | 95 | 101 | | A2 | 0.07 | 0.11 | -0.07 | 0.07 | 96 | 100 | | Avg w/o B2 | 0.06 | 0.08 | -0.01 | 0.06 | 96 | 100 | +------------+------+------+-------+------+------+------+ 2) Speed 4 Overall performance (all three features combined) +------------+-------+-------+-------+-------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+-------+-------+-------+-------+------+------+ | A1 | -0.01 | -0.09 | -0.01 | -0.02 | 90 | 101 | | A2 | 0.14 | 0.33 | 0.18 | 0.14 | 89 | 101 | | Avg w/o B2 | 0.09 | 0.20 | 0.12 | 0.09 | 89 | 101 | +------------+------+-------+-------+-------+------+------+ ```
…5103) Extend the intra-luma IST/STX pre-skip and post-trellis gates to cover eob == 0 in addition to the existing eob == 1 (DC-only) case. The eob == 0 branch kills candidates only when stx > 0, so a non-DCT primary with stx == 0 that quantizes to all-zero can still win via skip coding. Add a pre-quant L-infinity gate that predicts primary FP-quant eob = 0 directly from the post-primary coefficients (max |coeff| below the FP quantizer kill threshold min(dq0, dq1) >> (4 + log_scale)). When fired, av2_quant + av2_optimize_b + cost_coeffs are skipped and the same eob_found / DCT_DCT propagation as the post-quant gate is applied. Scoped to intra Y, stx == 0, non-DC-only, IST-enabled, non-QM blocks. Gated behind a new tx_sf.prune_intra_ist_stx_by_zero_eob flag, off by default and enabled at speed >= 1. Encoding at speed 0 is unchanged from the baseline. STATS_CHANGED Anchor: commit dd84ac9 Speed 1 (cpu-used=1): FG16 CTC (33 frames, RA) ``` +------------+-------+-------+-------+-------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+-------+-------+-------+-------+------+------+ | A1 | 0.06 | -0.20 | -0.34 | 0.02 | 98 | 100 | | A2 | 0.09 | -0.18 | -0.12 | 0.07 | 97 | 100 | | A3 | 0.05 | 0.58 | 0.48 | 0.07 | 96 | 99 | | A4 | 0.09 | 0.22 | -3.02 | -0.01 | 97 | 99 | | A5 | -0.12 | -2.35 | 2.22 | -0.12 | 97 | 101 | | B1 | 0.14 | -0.94 | -0.28 | 0.09 | 97 | 101 | | B2 | -0.48 | -0.71 | 0.30 | -0.46 | 97 | 97 | | Avg w/o B2 | 0.07 | -0.32 | -0.24 | 0.04 | 97 | 100 | +------------+-------+-------+-------+-------+------+------+ ```
Make it properly handle cctx cases. Take out coeff dropout to be consistent with what's used in RD search stage. Re-do u plane if its eob is different from that's in RD search to avoid potential mismatch issues due to the change of cctx_type. Tested for A2 and A4 in speed 1 with 33 frames. No coding stats change observed.
The unit tests that exercised base-only implicit-ref-map decode of a multi-layer bitstream (the only path where a TIP-winning base-layer block would fail to reconstruct) were removed in 9f0dfe5 (Fix setting in multi_layers tests, AOMediaCodec#5116), so this encoder-side workaround is no longer needed.
Skip process_compound_inter() function for opfl modes. Tests - cpu-used =1 Anchor: AOMediaCodec@6e71b0c Results (RA): A1 - 17 frames A2 - 33 frames <img width="1066" height="101" alt="MR7" src="https://github.com/user-attachments/assets/7e9e4bce-1065-4d28-84bc-a8df8f3479a7" />
…search (AOMediaCodec#5125) Enable winner transform search Add small fix for winner transform Disable 4 way and 5 way transform in rough search and enable in winner Set winner transform candidates to three Disable existing winner transform speed features. Tests - cpu-used =1 Anchor: 6e71b0c Results (RA): A1 - 17 frames A2 - 33 frames <pre> +---------+-------+--------+-------+-------+----------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +---------+-------+--------+-------+-------+----------+----------+ | A1 | 0.13% | 0.01% | 0.20% | 0.13% | 93.6% | 100% | | A2 | 0.17% | -0.15% | 0.06% | 0.15% | 92.3% | 100% | +---------+-------+--------+-------+-------+----------+----------+ </pre>
Squash merge branch 'leo/inter-sdp-mr' into 'avm-encoder-development' Add a speed level for inter sdp fast method: 1) Early terminate inter-sdp when there is no intra coded blocks in the this region after searching mixed intra and inter region; 2) Early terminate inter-sdp when more than half of the blocks are inter -coded; 3) Early terminate inter-sdp when the best block partitioning is partition-none STATS_CHANGED cpu-use-1 is at commit 6e71b0c is used as anchor. A1 - 17 frames (0.02% loss with 95.3% run-time) A2 - 33 frames (0.09% loss with 94.6% run-time) ``` +---------+-------+--------+-------+-------+----------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +---------+-------+--------+-------+-------+----------+----------+ | A1 | 0.01% | 0.17% | 0.11% | 0.02% | 95.3% | 100% | | A2 | 0.07% | 0.14% | 0.42% | 0.09% | 94.6% | 100% | +---------+-------+--------+-------+-------+----------+----------+ ```
Enable winner motion mode. Based on tid threshold, choose to perform six parameter warp searches in either winner mode or rough mode. Tests - cpu-used =1 Anchor: AOMediaCodec@6e71b0c Results (RA): A1 - 17 frames A2 - 33 frames ``` +---------+--------+--------+--------+--------+----------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +---------+--------+--------+--------+--------+----------+----------+ | A1 | -0.01% | 0.02% | 0.08% | -0.01% | 98.2% | 100% | | A2 | 0.01% | -0.07% | -0.01% | 0.01% | 98.4% | 101% | +---------+--------+--------+--------+--------+----------+----------+ ```
…nemv (AOMediaCodec#5134) Tested on 962f1c4 at speed 1, RA 33 frames +------------+-------+-------+-------+-------+-------+ | Class | Y | Cb | Cr | wAvg | Enc% | +------------+-------+-------+-------+-------+-------+ | A1 | +0.06 | +0.01 | +0.07 | +0.06 | 97.06 | | A2 | +0.08 | -0.08 | +0.03 | +0.07 | 96.94 | | Avg | +0.07 | -0.05 | +0.04 | +0.06 | 96.98 | +------------+-------+-------+-------+-------+-------+
Also added unit test with rtc mode Default remains GOOD quality mode.
This has two contributions Contribution 1 Block size based partition depth reduction for uneven 4 way Q index threshold is applied only to key frames. Apply separate partition depth for uneven 4 way partition from H - partition Contribution 2 Prune mv using best precision Do not prune one pel when maximum precision is quarter pel. Results: Anchor: 1d2febc Test: Contribution 1 + Contribution 2 ``` +---------+--------+-------+-------+-------+----------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +---------+--------+-------+-------+-------+----------+----------+ | A1 | -0.02% | 0.24% | 0.21% | 0.00% | 97.8% | 100.5% | | A2 | 0.10% | 0.15% | 0.03% | 0.09% | 95.8% | 99.8% | +---------+--------+-------+-------+-------+----------+----------+ ``` Test: Contribution 1 ``` +---------+--------+-------+--------+--------+----------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +---------+--------+-------+--------+--------+----------+----------+ | A1 | -0.02% | 0.05% | -0.14% | -0.02% | 99.0% | 100.5% | | A2 | 0.06% | 0.09% | 0.19% | 0.07% | 97.6% | 99.3% | +---------+--------+-------+--------+--------+----------+----------+ ```
This patch refactors av2_model_rd_curvfit() by consolidating two interp_cubic() calls for computing rate and distortion. Also, the relevant SSE2 implementation for the combined interp_cubic() function along with its corresponding unit test is added. The SSE2 implementation achieves ~1.21x speedup compared to the scalar C version. (adopted from libaom: https://aomedia-review.googlesource.com/c/aom/+/211441) Also, unnecessary calls to the function get_dqv() in update_coeff_general(), update_coeff_simple(), and update_coeff_eob() are removed. Test results (RA) Anchor: commit 165345b A1 - 17 frames A2 - 33 frames ``` -------------------------------------- Speed Class Encoder Instruction Count Reduction (%) ------------------------------------- 0 A2 0.32 A1 0.23 ------------------------------------- 1 A2 0.31 A1 0.21 ------------------------------------- 2 A2 0.19 A1 0.10 ------------------------------------- 3 A2 0.26 A1 0.11 ------------------------------------- ```
Tweak the speed feature 'restrict_tx_partition_type_search'. Anchor: 41fc9bf Test condition: CTC, RA, 33 frames, speed 1 ``` +---------+--------+--------+--------+--------+----------+ | Summary | Y | U | V | YUV | Enc-time | +---------+--------+--------+--------+--------+----------+ | A1 | 0.07% | 0.04% | 0.04% | 0.07% | 96.91% | | A2 | 0.05% | 0.03% | 0.22% | 0.05% | 96.22% | |avg wo b2| 0.06% | 0.04% | -0.02% | 0.06% | 96.24% | +---------+--------+--------+--------+--------+----------+ ``` STATS_CHANGED
) Introduce a speed feature for speed >= 1 to 1. Allow only one same reference compound (0,0) and disallow (1,1) 2. Define a priority array for compound reference pairs based on selection frequency, and only allow the top 2 in the array for all compound modes other than NEAR_NEARMV and NEAR_NEARMV_OPTFLOW The macro SAME_REF_COMPOUND_PRUNE in entropymode.h is renamed and moved to encoder.c because it is only used by the encoder. RA 33 frames, speed 1 Anchor: 8fbe562 +---------+--------+--------+--------+--------+----------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +---------+--------+--------+--------+--------+----------+----------+ | A1 | +0.01% | +0.06% | -0.05% | +0.01% | 97.9% | 100.0% | | A2 | +0.04% | +0.00% | +0.35% | +0.04% | 97.4% | 100.6% | | Avg | +0.03% | +0.02% | +0.23% | +0.03% | 97.6% | 100.4% | +---------+--------+--------+--------+--------+----------+----------+
This change adds a macro `MAXQ_FOR_GIVEN_BIT_DEPTH()` and replaces the ternary operators used to compute max-qp using the macro. No stats changed.
Add a tid based fast method for AMVD under newmv mode. The results are tested on top of commit 5c29069. Class A1 and A2 results are llisted below. ``` +---------+--------+--------+--------+--------+----------+ | Summary | Y | U | V | YUV | Enc-time | +---------+--------+--------+--------+--------+----------+ | A1 | 0.07% | -0.12% | -0.03% | 0.05% | 98.2% | | A2 | 0.06% | 00.14% | -0.24% | 0.05% | 97.9% | +---------+--------+--------+--------+--------+----------+ ``` STATS_CHANGED
Skip AMVD modes for NEAR NEWMV, NEW NEARMV, NEAR NEWMV OPFL and NEW NEARMV OPFL Skip AMVD modes when tid level is greater than 3 Results: Anchor: 0d25d58 ``` +---------+-------+-------+--------+-------+----------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +---------+-------+-------+--------+-------+----------+----------+ | A1 | 0.06% | 0.09% | 0.17% | 0.07% | 97.6% | 99.7% | | A2 | 0.04% | 0.13% | -0.04% | 0.04% | 97.1% | 101.3% | +---------+-------+-------+--------+-------+----------+----------+ ```
Search warp interintra in rough mode when wrl index is greater than or equal to 2 Search warp interintra in winner mode for remaing wrl indices if warp interintra is selected in rough mode Search warp interintra in winner mode for warpmv with mvd flag if warp interintra is selected in rough mode with warpmv with mvd flag off Anchor: ef64fa8 Results: ``` +---------+-------+--------+-------+-------+----------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +---------+-------+--------+-------+-------+----------+----------+ | A1 | 0.07% | -0.01% | 0.23% | 0.07% | 97.6% | 100.1% | | A2 | 0.05% | -0.03% | 0.28% | 0.06% | 96.8% | 99.0% | +---------+-------+--------+-------+-------+----------+----------+ ```
- Abstract distortion calculation into helper function get_tx_blk_distortion() returning RD_STATS. - Introduce get_ist_max_set_id() and get_ist_set_id() helper functions. - Improve readability of prune_rectangular_tx_type() by removing redundant tx_mask_32[][] lookup table. - Eliminate redundant call to av2_get_ext_tx_set_type() in search_tx_type() by reusing txfm_param.tx_set_type. - Add early loop exit for FSC mode when transform dimensions exceed FSC_MAXWIDTH / FSC_MAXHEIGHT on luma plane. No stats changed.
* MLP intra mode pruning Replace HOG-based directional mode pruning with an MLP that predicts the top-5 most likely luma intra base modes per block. The 4-layer network (104->128->64->32->13) takes an 8x8 downsampled source patch, neighbor modes, QP, block size, and a 32-bin HOG histogram as inputs. Active in both all-intra (AI) and inter (RA/LD) frames; falls back to HOG for 4x4, 4x8, and 8x4 blocks. Enabled at speed >= 1 via sf.intra_sf.intra_pruning_with_mlp. Anchor: commit b4e5915 Performance of this change in isolation, Speed 1 (cpu-used=1), FG16 CTC (33 frames): AI: ``` +------------+-------+-------+-------+-------+---------+---------+ | Class | Y | Cb | Cr | wAvg | EncWall | DecWall | +------------+-------+-------+-------+-------+---------+---------+ | A1 | 0.14 | 0.05 | -0.02 | 0.13 | 91 | 101 | | A2 | 0.17 | 0.06 | -0.03 | 0.15 | 90 | 99 | +------------+-------+-------+-------+-------+---------+---------+ | Avg w/o B2 | 0.17 | -0.02 | 0.07 | 0.16 | 90 | 101 | +------------+-------+-------+-------+-------+---------+---------+ ``` RA: ``` +------------+-------+-------+-------+-------+---------+---------+ | Class | Y | Cb | Cr | wAvg | EncWall | DecWall | +------------+-------+-------+-------+-------+---------+---------+ | A1 | 0.13 | 0.17 | -0.58 | 0.09 | 97 | 100 | | A2 | 0.11 | 0.04 | 0.05 | 0.10 | 96 | 100 | +------------+-------+-------+-------+-------+---------+---------+ | Avg w/o B2 | 0.03 | 0.03 | -0.32 | 0.01 | 96 | 100 | +------------+-------+-------+-------+-------+---------+---------+ ``` STATS_CHANGED
- partition_allowed_*, prune_partition_* fields are grouped into one array and made accessible via partition_type No stats changed Change-Id: I9983d117e36805b9c936038cdcdbd22c2fc23413
This patch avoids extra stack buffer size for `process_ref` structure and uses the correct size `MFMV_STACK_SIZE` in `av2_setup_motion_field()`. This aligns with the maximum allowed size check in `check_and_add_process_ref()`.
This patch refactors `check_and_add_process_ref()` by removing `checked_count` and its associated code as `process_count` serves the same purpose.
Initially, `ProcessRefTMVP->type` was introduced to indicate whether to use side projection (type 0) or target frame projection (type 1). Later, the logic was simplified and type 0 is only used for every case. As a result, `motion_field_projection_start_target()` is entirely unused. This patch cleans up these unused code.
The recent removal of `motion_field_projection_start_target()` causes GCC compiler to inline `motion_field_projection_side()` into `av2_setup_motion_field()`, increasing the decoder instruction count by ~0.5%. Marking the function noinline prevents this issue. Clang compiler shows neutral impact for this change.
Introduce 3 EOB levels adaptive tx search pruning. eob_level = 0: eob >= max_eob / 4 eob_level = 1: max_eob / 12 <= eob < max_eob / 4 eob_level = 2: eob < max_eob / 12 Anchor: AOMediaCodec@0d25d58 Tests: cpu-used =1, RA A1 - 17 frames A2 - 33 frames ``` +---------+--------+--------+--------+--------+----------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +---------+--------+--------+--------+--------+----------+----------+ | A1 | 0.09% | -0.06% | -0.99% | 0.03% | 96.4% | 99.5% | | A2 | 0.09% | -0.35% | -0.20% | 0.06% | 97.7% | 99.6% | +---------+--------+--------+--------+--------+----------+----------+ ```
Make the C and AVX2 code have the same behavior. Add unit test to check the two match.
MLP-based partition pre-screener
Add an MLP-based partition mode pre-screener for the encoder's partition
search. A small 3-layer network (38 inputs, hidden dims 64/32, 4
outputs) predicts NONE/HORZ/VERT/SPLIT per block from a 32-bin HOG
histogram over the source luma plane plus 6 scalar features (source
variance, block size, QP, RD cost of the NONE partition, and the winning
partition type of the above/left neighbors). Separate models are trained
for intra-only and inter frames. When the model confidently predicts
NONE, the weaker of the HORZ/VERT rectangular directions is pruned from
the RD search. Enabled at speed >= 1 via
sf.part_sf.partition_pruning_with_mlp.
Anchor: commit 1d2febc
Performance of this change in isolation, Speed 1 (cpu-used=1), FG16 CTC
(33 frames):
AI:
```
+------------+------+-------+-------+------+------+
| Class | Y | Cb | Cr | wAvg | Enc% |
+------------+------+-------+-------+------+------+
| A1 | 0.14 | 0.05 | -0.01 | 0.13 | 90 |
| A2 | 0.17 | 0.05 | -0.02 | 0.16 | 90 |
+------------+------+-------+-------+------+------+
| Avg w/o B2 | 0.17 | -0.02 | 0.08 | 0.16 | 90 |
+------------+------+-------+-------+------+------+
```
RA:
```
+------------+------+------+-------+------+------+
| Class | Y | Cb | Cr | wAvg | Enc% |
+------------+------+------+-------+------+------+
| A1 | 0.12 | 0.23 | -0.40 | 0.10 | 96 |
| A2 | 0.07 | 0.24 | 0.36 | 0.08 | 95 |
+------------+------+------+-------+------+------+
| Avg w/o B2 | 0.13 | 0.42 | -0.09 | 0.12 | 95 |
+------------+------+------+-------+------+------+
```
STATS_CHANGED
This change reduces duplicate code in functions that recurse through pc trees. No stats changed
Anchor: 6d405a3 Test condition: CTC, RA, 33 frames, speed 1 +---------+--------+--------+--------+--------+----------+ | Summary | Y | U | V | YUV | Enc-time | +---------+--------+--------+--------+--------+----------+ | A1 | 0.01% | -0.03% | 0.08% | 0.01% | 99.14% | | A2 | 0.02% | 0.05% | 0.03% | 0.02% | 97.98% | |avg wo b2| 0.02% | 0.04% | 0.04% | 0.02% | 97.69% | +---------+--------+--------+--------+--------+----------+ STATS_CHANGED
Introduced an encoder option: "enable_low_complexity_decode" (LC mode) to target reduced decoding complexity. This mode is disabled by default. Also added related API control for ongoing LC mode development.
1. Cleanup the unused sf 'enable_interinter_diffwtd_newmv_search` and its associated code 2. Cleanup the code and comments related to COMPOUND_DISTWTD No Stats Changed
This feature considers the distortion reduction of deblocking filter for partition research. -- The deblocking filter is applied to a non-split partition after the best mode is picked for the current block. The distortion improvement of the deblocking filter is used to update the best RD cost of the current none-split partition block. This encoder side only feature provides the coding gain with slight encoding time increase. Should be a good trade-off for low speed level. Results of 33 frame (cpu1), a1 (missing 3 sequences) and a2 (missing 1 sequence) are as follows: +---------+--------+-------+-------+-------+----------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +---------+--------+-------+-------+-------+----------+----------+ | A1 | -0.62% | -0.14% | -0.49% | -0.58% | 102.06% | 100.82% | | A2 | -0.39% | +0.15% | -0.70% | -0.38% | 102.37% | 100.56% | +---------+--------+-------+-------+-------+----------+----------+
2e7096f to
78d9f4c
Compare
|
@yeqing-wu : I'm running I'm getting some assertion failures locally for RTC tests: @jianj-g : was this fixed on av2-enc later on? |
Yes this should be fixed (also RtcTest is non-large now so it gets run nightly / each PR) |
I'm aware. But we are only pulling commits from that branch up to a certain commit. Later commits will not be pulled in this iteration. Can you point to commit that fixed this issue? |
|
Hi @urvangjoshi, I've run the following tests:
As of this afternoon that gives 4,942 bitstreams in total, all of which decode successfully with the decoder binary built from the av2-normative branch. So once we resolve the assert failure you found in the RTC tests, this PR should be ready Thanks |



Please don't squash — we need to preserve the commit history, so rebase only.